E2E-BPF microscope: extended depth-of-field microscopy using learning-based implementation of binary phase filter and image deconvolution

Several image-based biomedical diagnoses require high-resolution imaging capabilities at large spatial scales. However, conventional microscopes exhibit an inherent trade-off between depth-of-field (DoF) and spatial resolution, and thus require objects to be refocused at each lateral location, which is time consuming. Here, we present a computational imaging platform, termed E2E-BPF microscope, which enables large-area, high-resolution imaging of large-scale objects without serial refocusing. This method involves a physics-incorporated, deep-learned design of binary phase filter (BPF) and jointly optimized deconvolution neural network, which altogether produces high-resolution, high-contrast images over extended depth ranges. We demonstrate the method through numerical simulations and experiments with fluorescently labeled beads, cells and tissue section, and present high-resolution imaging capability over a 15.5-fold larger DoF than the conventional microscope. Our method provides highly effective and scalable strategy for DoF-extended optical imaging system, and is expected to find numerous applications in rapid image-based diagnosis, optical vision, and metrology.


Introduction
Microscopic imaging systems can only produce a clear image of an object within a limited depth range, known as depth-of-field (DoF).The DoF defines the depth range of an object that can be sharply imaged by a given optical imaging system, and is determined by the operating wavelength, effective focal length and aperture size of the imaging lens.In many biomedical imaging applications, such as in cytometry 1,2 , histology 3 , and endoscopy [4][5][6] , high-resolution imaging over a large spatial scale is often desired; for instance, a pathological examination is typically performed with a high numerical-aperture (NA) objective to visualize cellular and subcellular features of tissue specimens, but it is accompanied by limited fieldof-view (FoV) and DoF.Therefore, to image large-area pathological/cytology slides, either objects or imaging optics should be scanned and refocused repetitively, which is costly and labor-intensive.To enhance the DoF, various strategies have been explored over the past few decades.A simple solution would be to reduce the aperture size of the detection system as the DoF increases with 1=NA 2 ; however, this inevitably causes a loss of light throughput and information capacity.Wavefront coding, combined with dedicated deconvolution methods, provides a convenient and effective route for enhancing DoF performance 7 .Various pupil filters, such as the cubic phase mask (CPM) 7 , sinusoidal 8 , logarithmic 9 , tangent 10 phase filters and hybrid refractive-diffractive structures have been introduced for the DoF-extension and to correct for some forms of aberrations 11 .However, implementing such complex and continuous phase structures requires either expensive phase-modulating devices (e.g., spatial light modulators) or sophisticated manufacturing methods (e.g., e-beam or multi-step lithography).
Binary phase filters (BPF), composed of concentric rings with phases of 0 and π (i.e., 1, −1 in amplitude), have recently received considerable attention as DoFextension elements owing to their simple topology and ease of manufacturing.As the object information travels through a carefully designed BPF, the resulting images can be tuned to be invariant over the desired depth range, while maintaining a high lateral resolution.In addition, owing to its discrete 0-π phase topology, BPF allows relatively simple manufacturing processes such as photolithography and thin-film deposition, making them suitable for mass production.Consequently, various BPF design methods for focus-or DoF-extension have been suggested in recent years, including exhaustive search [12][13][14][15][16][17][18][19] , analytical solutions [20][21][22][23] , and various types of optimization algorithms [24][25][26][27] .Despite these efforts, the DoF-extension performance of BPFs has not been fully explored.One of the main reasons is that, while its performance improves with an increasing number of rings 28 , developing BPF designs with more than five concentric rings is extremely challenging and computationally expensive due to the complexity of the non-linear equations involved.For multi-annulus binary filter designs involving exhaustive searching algorithms, the processing time increases exponentially with the number of concentric rings 16 .Particle swarm optimization (PSO) algorithms 29 , which are known to be effective in solving non-linear multi-dimensional problems, have been employed to design BPF by exploring the vast design space in multi-annulus binary optical elements 28,[30][31][32][33] .However, PSO-based algorithms require a number of preset design parameters, and for a design task involving many parameters, the solution space is expected to grow exponentially.Moreover, PSO tends easily to fall into local optimum in highdimensional space and has a low convergence rate in the iterative process 34 .
Here, we present a DoF-extension computational imaging platform enabled by an end-to-end optimized BPF and image reconstruction (E2E-BPF microscope).To develop BPF designs with no constraints on the number of rings, we adopted a deep learning-based endto-end framework to jointly design the DoF-extension BPF and optimize the relevant imaging reconstruction network with a large number of datasets.The deep learning-based BPF design is enabled by introducing a penalization function in the network, which involves differentiable design variables that converge to binary states through epochs.The learned BPF was inserted into an optical microscope to produce a depth-invariant point-spread function (PSF) over the extended DoF.The resultant images were then fed into the jointly learned deconvolution network to produce highresolution and high-contrast images over the extended DoF.We demonstrate high-resolution, high-contrast imaging capability over a >15.5×DoF of our E2E-BPF platform through numerical simulations and experiments with fluorescent beads.The biological viability of our method was further demonstrated by imaging cellular specimens and a large-scale mouse kidney tissue section stained with fluorescent dyes with no refocusing.

E2E-BPF microscope: physics-informed, learning-based BPF design and image deconvolution
DoF for an optical microscope with a circular aperture is determined as 35 : where n medium is the refractive index of imaging medium, λ is the wavelength of light, and NA is the numerical aperture of the objective.M is the magnification factor of the microscope, and e denotes the pixel pitch of image sensor.The DoF clear in our experimental setup (33×/0.75NA)was estimated to be 1.19 μm.Our goal is to obtain high-resolution images over the extended DoF with jointly optimized front-end binary-phase optics and the back-end reconstruction algorithm (Fig. 1).We achieved this using end-to-end training of the BPF design and neural network as a joint optimization problem.The design process involves an evolution of both the phase filter design (i.e., the phase of each ring in the BPF parameterized by ϕ) and the post-processing algorithm N ðÁÞ (i.e., trainable hyperparameters in N ðÁÞ parameterized by W net ).Our proposed architecture accomplishes supervised learning using a set of groundtruth images I T to educate and evolve hardware/ software variable parameters.This design pipeline and backpropagation procedure are shown in Fig. 2. The architecture consists of two major components: (1) a differentiable imaging model with a BPF to be designed, which takes in input ground-truth image and corresponding depth information ψ (see Eq. 7 for the definition of ψ) and outputs an intermediate image I predicted by the forward imaging model, and (2) a deconvolution neural network to produce a highresolution, high-contrast image from the intermediate image.The optical layer simulated the image formation of a microscope with a phase filter in its pupil plane.Given a phase filter and an object with a certain defocus distance, we obtain an intermediate image (I) by convolving the ground-truth object information (I T ) and the corresponding PSF.We defined the design variables as the phase values of K concentric annular regions, parameterized by K phase values ϕ ¼ ðϕ 1 ; ϕ 2 ; ; ϕ K Þ.In our analysis, it was set to K = 64 for a desired DoF of 16× that of clear aperture, as it provided a numerically accurate system response while minimizing computational cost.The same BPF design was derived for a larger K (e.g., K = 128) when using the same initial conditions.Detailed analysis on the number of rings for a desired DoF is provided in Sec. 1 in Supplementary.To induce the phase value to the binary states during the learning stage, a differentiable penalization function PðÁÞ was introduced within the end-toend optimization framework (Fig. 2).The penalization function was designed to have saddle points on (-π, 0, π) to facilitate the convergence of the phase value to those values at the end of training.Finally, the BPF was obtained by taking the absolute value and threshold of the phase values.Note that the proposed penalization function accepts and produces continuous values in the range [-π, π], and the phase filter can be initialized with a generalized pupil function (e.g., Zernike functions).In our ablation study, we found that the use of phase axicon and spherical aberration as the initial conditions markedly improved optimization performance (See Sec. 2 of Supplementary).The forward imaging model performs imaging in a wide-field fluorescence microscope with a phase filter in its pupil plane to obtain the intermediate image (I).Then, U-Net, a widely used neural network for solving such deconvolution problems 36 , was trained to obtain the final image, which is compared against the ground-truth image (I T ).Both the reconstruction network and phase values of the BPF are updated to minimize the end-to-end loss function L E2E through a gradient-descent method.Our optimization problem is stated as: where the first term L RMSE ðÁÞ evaluates the difference between the post-processed image N I ϕ; ψ À Á ; W net À Á and the ground-truth I T , and the second term L BPF ðÁÞ is a BPF feature loss that enforces the phase values of BPF rings to the binary states.α is a penalty parameter that controls the relative weight of the two terms, and it is updated through the epochs.Details of the algorithm and definitions of the loss functions are provided in the "Methods" section.

Numerical experiments and validation
First, we validated the DoF-extension performance of the E2E-BPF microscope through numerical experiments under the same conditions as those of our experimental setup.To this end, we define the DoF of a microscope with a phase filter A (DoF A ), as the axial range over which the structural similarity index measures (SSIMs) of the Here, I T and Î are the ground-truth information and the reconstructed image for the object defocused by z, respectively, and DoF A denotes the DoF obtainable with pupil A and the corresponding reconstruction U-Net.SSIM thr is the threshold SSIM value, which can be set by the user.In our implementation, we set SSIM thr to be the SSIM value at DoF clear = 1.19 μm for the clear aperture, which is SSIM thr ¼ 0.900 for our test dataset 37 .Having defined the DoF, we examined and compared the performance of the E2E-BPF microscope against those from microscopes with a clear aperture and CPM.The effective NAs were identical for all three configurations.For evaluation, images of Lenna and fluorescent tissue sections 38 were used as ground truths, which were imaged using a microscope with a given pupil filter.We then trained the reconstruction U-Nets for each imaging condition, except for the images with the clear aperture.Figure 3 presents representative images at various defocus distances obtained by microscopes with clear, E2E-BPF and CPM filters in the pupil plane.Note that the defocus distances were normalized with DoF clear =2.One can easily observe that E2E-BPF and CPM provide highcontrast, high-resolution images over much larger depth ranges, whereas the image quality for the clear aperture degrades rapidly for defocus distances exceeding DoF clear (1.19 μm).We evaluated the root mean square error (RMSE) and SSIM values as a function of the normalized axial defocus distance (see Methods for the definition of image evaluation metrics) with 820 image patches from independent datasets (Fig. 3d).Consistent with the qualitative observation in Fig. 3a-c, E2E-BPF and CPM microscopes offered much higher SSIM and smaller RMSE values over a much larger DoF (19.93 μm), as compared with those of the microscope with clear aperture (1.19 μm).Compared to CPM, E2E-BPF provided higher SSIM and smaller RMSE values over the entire defocus range.In specific, as shown in Fig. 3d, SSIM values above SSIM thr (0.900) could be obtained up to z= ±9.96 μm for E2E-BPF (16.74× larger DoF based on Eq. 3), while those were limited to z= ±7.57μm for CPM.The mean SSIM values of E2E-BPF and CPM over the entire DoF range were found to be 0.947 and 0.904, respectively.One can also note that the in-focus SSIM value of CPM (0.894) was smaller than SSIM thr , while those for clear aperture and E2E-BPF were found to be 0.922 and 0.944, respectively.We further examined the DoF-extension performance of E2E-BPF against other prior pupil designs (Table 1).We used the same datasets in Fig. 3 (i.e., images of Lenna and fluorescent tissue section as the groundtruths), and trained the reconstruction U-Nets for each imaging condition.RMSEs and SSIMs were computed for the resultant images, and the average RMSE and SSIM were evaluated for all the images in the test dataset within 20 μm DoF.The DoF-extension ratio was calculated as the ratio of DoF A to DoF clear .As shown in Table 1 and Fig To begin, the phase filter is initialized with a continuous axi-symmetric function (e.g., axicon) and penalized by a nonlinear function that is designed to enforce the phase value in each ring to the binary states through the training process.The imaging model then predicts the image, which is then fed into the U-Net-based image reconstruction network to obtain the network output.This network output is compared against the ground-truth image, and optimization is performed to minimize the difference through a gradient-descent method deconvolution network based on the loss function set with image metrics over a large number of images.While BPFs combined with various deconvolution algorithms 30,39 and reconstruction networks 40 have been proposed, E2E-BPF utilizes a significantly larger number of design variables, and thus the algorithm can explore vast spaces to obtain optimal BPF designs that produce depth-invariant PSFs over the desired DoF range.The  jointly optimized deconvolution network further denoises and processes the acquired images to yield high-resolution, high-contrast images.

Experimental performance evaluation: fluorescence microspheres
We then experimentally validated the DoF-extension performance of the E2E-BPF microscope by imaging green fluorescent beads (PS-Speck Microscope point source kit 7220, Molecular Probes, USA).An E2E-BPF designed with a phase axicon as the initial condition was fabricated using photolithography, and inserted into the pupil plane of a custom-built fluorescence microscope (see Methods).The fluorescence beads were sufficiently smaller than the diffraction-limited resolution of the microscope (0.75NA); thus, the image of a single bead could be considered as the PSF.We acquired images of the beads with and without the E2E-BPF in the microscope, as the monolayered beads were scanned along the optical axis in steps of 0.1 μm in the range of -12 μm to 12 μm.At each depth, we acquired 10 frames with a 100 ms exposure time, and averaged and subtracted the background to reduce noise.The images were reconstructed using the U-Net jointly optimized by the numerical simulation.Figure 4a, b shows representative images of a fluorescent bead acquired at various defocus distances with standard and E2E-BPF microscopes.For visual clarity, all the images were normalized by the peak value of the image at z= 0 μm for each case.In the case of the images from the standard microscope (i.e., microscope with a clear aperture), the beads became immediately blurred as they were displaced by 0.6 μm from the focal plane of the objective lens.In contrast, the E2E-BPF microscope produced high-resolution, high-contrast images of the beads over the depth range of −9.5 μm to 9.5 μm.We evaluated the full widths at half-maximum (FWHMs) of the PSFs at various depths (Fig. 4c).We performed Gaussian fitting on the intensity profiles of the bead images, and computed the FWHMs.The in-focus FWHM of the standard microscope was measured to be Imaging experiments were carried out using the same standard and E2E-BPF microscopes as described in the previous section.Figure 5a presents an image of the BPAE cells captured by the E2E-BPF microscope.Two regions in the imaging FoV, marked with orange and green dotted boxes were examined at various defocus distances (Fig. 5b,  c).One can see that for the defocused images from the standard microscope (with clear aperture), both the image quality and the SSIM values decreased dramatically.By contrast, the E2E-BPF microscope produced the highcontrast images with high SSIM scores at all depths.Specifically, all the images from the E2E-BPF microscope featured SSIM values larger than 0.9 in the range from −9 μm to 9 μm, and the mean SSIM value was found to be 0.95.In contrast, the mean SSIM values from the standard microscopy images were measured to be 0.54.
The insets in Fig. 5b, c show the intensity profiles along the solid lines in Fig. 5a.Notably, the images from E2E-BPF microscope feature high-contrast (or high modulation depth) over the extended DoF, while the standard microscope provides high-contrast images only in the focal plane (z= 0 μm).Specifically, in Fig. 5b, c, the contrasts of the images from standard microscope were 0.99 and 0.98 at the focal plane, but decreased to 0.75 and 0.83 at the defocus distance of 9 µm, respectively.The mean contrast values of the images from the standard microscope over the range of −9 µm to 9 µm were found to be 0.83 and 0.88, while the mean contrast values of images from E2E-BPF microscope were 0.97 and 0.96, respectively.

Experimental result of E2E-BPF microscope: multicolor fluorescent imaging
We further imaged a 16-μm thick mouse kidney tissue section stained with multiple fluorescent markers (Fluo-Cells® prepared slide #3 (F24630)) to demonstrate the utility of the E2E-BPF microscope for large-area, high-throughput imaging applications.Mouse kidney tissue was stained using a combination of three fluorescent dyes: DAPI (blue) to stain the DNA, AF488 (green) to label the tubules, and AF568 (red) to visualize the F-actin filaments.Imaging experiments were performed using the same standard and E2E-BPF microscopes as in the previous section, and we employed the same image reconstruction networks for all images obtained.
Figure 6a presents a whole slide image of the mouse kidney tissue captured using the E2E-BPF microscope without serial refocusing.The image was produced by integrating 589 individual frames, each with dimensions of 2048 × 2048 pixels.The frames were stitched together with a standard image stitching algorithm 41 with an overlap of 10% to ensure seamless integration of the frames.After the whole image was constructed, it was divided into small patches of 576 × 576 pixels, which were then inputted to the reconstruction U-Net network (See Sec. 4 of Supplementary for detailed information on U-Net).The output of the U-Net network was then reassembled into the whole slide image through a mosaic algorithm.The U-Net was capable of post-processing at a speed of 0.01 s/576 × 576 pixel patches.The total image acquisition and processing time of the E2E-BPF microscope was measured to be 30 min, which is more than 15.5 times shorter than that of a standard microscope with serial refocusing.The two regions in the image, marked with yellow dotted boxes, are shown in greater detail in Fig. 6b, d for the standard microscope and Fig. 6c, e for the E2E-BPF, respectively.Comparing the images obtained with the E2E-BPF and standard microscope, the images from standard microscope in Fig. 6b, d appear partially defocused and blurred due to its limited DoF (1.19 μm), which is much smaller than the thickness of the mouse kidney tissue section (~16 μm).In contrast, the E2E-BPF microscope provided all-infocus images, as demonstrated in Fig. 6c, e.This difference was even more pronounced when comparing the enlarged images indicated by the white dotted box in Fig. 6b-e.Enlarged images in Fig. 6b1, c1 are glomerular regions where tubules are intricately entangled, and Fig. 6d1, e1 indicate a glomerular region with relatively low density.Due to the three-dimensional arrangement of the glomerulus at various depths, the microscope with the limited DoF produced diffuse and low-contrast images, whereas the E2E-BPF microscope could clearly image tubular and nuclei structures indicated in green and blue, respectively.Figure 6b2, c2, d2, e2 show the enlarged images of the tubule and duct regions, respectively.The nuclear and cytoplasmic fluorescence signals at various depths led to a diffuse background in the standard microscope, while the E2E-BPF microscope could clearly image structures across various depths.To validate E2E-BPF imaging on mouse kidney sections, we conducted axial scanning and quantified the local image contrast for enlarged images shown in Fig. 6b1-e1, b2-e2 (Sec.5, Supplementary).The E2E-BPF microscope could resolve nuclei, tubules, and duct structures with a mean contrast of 0.95 for the defocus ranges considered.In contrast, the standard microscope produced partially focused images in the defocus range of 0 μm to 3 μm with a mean contrast of 0.83.
We further performed E2E-BPF imaging of 3D tumor spheroid of nominal thickness of 50 μm and compared its imaging performance against standard microscope.Details of tumor spheroid formation and imaging results are

Discussion
We presented a computational microscopy platform capable of high-resolution imaging of large-scale specimens over 15.5× larger DoF.We developed a datadriven, physics-informed, deep-learning architecture to jointly design and optimize a binary phase structure and image reconstruction network.We compared the imaging performance of our platform with previously reported phase filter designs and demonstrated its superior imaging performance.Experimental validations were also performed by imaging fluorescently labeled beads and tissue sections to demonstrate its validity in visualizing detailed structures across specimens without serial refocusing.
Compared to prior studies, several distinctive features should be noted in our platform: (1) Our method aims to obtain BPF designs rather than continuous phase filters.
The phase filters with continuous and complex functions are often found to be challenging to fabricate.Consequently, most relevant studies have used sophisticated fabrication methods, such as e-beam lithography and multistep photolithography, or employed active wavefront modulation devices (e.g., spatial light modulators), which are expensive and make the system bulky.In contrast, BPF is a transparent, two-state phase element; therefore, it is relatively easy to fabricate and offers amenability to mass production.Simple one-step photolithography or nanoimprinting can readily produce the designed BPF on a large scale.
(2) To the best of our knowledge, our method represents the first end-to-end deep learning-based implementation of BPF and image deconvolution.The design of binary structures in a DNN framework is challenging, as it is associated with the gradient computation of binary functions.Some BPF design studies detoured this problem by approximating a binary function with some continuous functions 40 .We tackled this problem by introducing a differentiable penalization function and BPF loss term in our network, which resolves the discontinuity problem, while facilitating convergence to binary states in the final BPF design.We believe that our method is a viable design methodology for deep-learning-based binary structures in various optical applications.(3) We experimentally demonstrate the large-DoF imaging performance of E2E-BPF microscope over a broad range of the visible spectrum.Jin et al. demonstrated 5× extended DoF performance on single-color fluorescence imaging 3 using a phase filter of continuous phase functions.Our method, on the other hand, provides much larger DoF-extension performance (15.5× compared with a clear aperture) with a binary phase filter, and demonstrated its imaging capability for both single-and multiple-color fluorescence imaging.Although the BPF was optimally designed for a single wavelength and aberration-free optical system, we demonstrated the robustness of E2E-BPF in DoF-extension to multicolor imaging.These features altogether suggest a great promise of our method in a wider range of applications in biomedical diagnosis and color vision, for example.Our design was performed in an aberration-free microscope using a single wavelength (center wavelength of the operating spectrum), and thus any discrepancies between our model and experimental settings may contribute to the degradation of DoF.Our experimental results for the mouse kidney section stained with three fluorophores demonstrated robustness of multicolor imaging in E2E-BPF platform.However, if the fluorescent molecules exhibit emission spectra far distant from the design wavelength, the imaging performance is expected to degrade (See Sec.7 of Supplementary).We numerically performed E2E-BPF imaging of 820 objects labeled with various fluorescent dyes (i.e., DAPI (blue), FITC (green), TRITC (red), and Cy7 (far-red)), which exhibit different emission wavelengths.The results indicate that E2E-BPF designed at 525 nm is robust to variations in emission wavelength of <110 nm, but if the spectral shift from the design wavelength exceeds 250 nm, the performance of the E2E-BPF microscope decreases.
One can consider the extension of our platform in various directions.For instance, one might incorporate the system aberration into our design framework to further enhance the image quality.The measurements of the system aberration can be performed, for example, by imaging isolated fluorescent particles across the 3D space of interest 42 .The PSFs can then be incorporated into the physical model to jointly optimize filter structure and deconvolution network.To demonstrate the viability of this aberration-informed E2E-BPF design, we performed numerical experiments (Sec.8, Supplementary), and found that the aberration-informed E2E-BPF design outperformed aberration-ignorant BPF design in terms of both DoF and image quality.Further, this aberration-informed design strategy can be extended to handle spatially-varying aberration in 3D microscopes.In this case, an axi-symmetric BPF may not be suitable for handling spatially-varying aberrations, and one may thus need to explore more design spaces and configurations (e.g., binary or continuous phase functions).In addition, aberrations derived from possible mismatch between nominal immersion liquid and samples of imaging, which is a major source of image degradation in high-NA (i.e., NA > 1) imaging systems, can be considered.This can potentially be addressed by incorporating more accurate scalar or vector beam propagation models (e.g., the Gibson & Lanni scalar model 43 ) that better describe these imaging characteristics into the proposed framework.
In our study, we set our desired imaging depth to be 16× that of clear aperture, and performed the design using 64 design variables.It should be noted that our method is capable of generating E2E-BPF platform with further DoF-extension.To achieve this, however, the number of design variables (i.e., the number of rings in BPF in our case) should be increased, which would markedly increase the computation and training times for the BPF design and deconvolution network.We indeed performed BPF design for 24× DoF-extension with 128 design variables, and obtained the BPF design with 22.08× DoF-extension (Sec.9, Supplementary).The design, however, required 2.5× longer computation time compared with the original 64-ring design.Moreover, the reduction of fluorescence intensity in the detector plane should be taken into account.Since BPF generates elongated PSFs in the detector region, the energy is distributed over the depth, which results in the decrease in the measured fluorescence signal.This feature has been noted by prior publications 24,44 .Depth-resolved, high-resolution imaging over extended 3D space can also be considered as a potential extension of our platform.The design framework can be tailored to produce PSFs that vary distinctively with emitter locations, and jointly optimized neural network generates high-resolution images over the entire 3D space 45,46 .Implementation of such microscopes may involve the exploration of various forms of amplitude 47,48 , phase 49,50 or hybrid filter structures with continuous and multi-step phase functions.
In terms of applications, one of the potential applications is its utility in light sheet fluorescence microscopy, which calls for large-DoF and high light efficiency.A BPF can be designed to generate sharp and elongated excitation light sheet or focus on the illumination path 51 .The resultant fluorescence emission from a large 3D sample can be detected though our E2E-BPF platform, allowing for high light-throughput, high-resolution volumetric imaging of fluorophores without re-focusing.In our experiments, we did not observe any notable photodamage and photobleaching in longitudinal E2E-BPF imaging (Sec.10 of Supplementary).The E2E-BPF platform is also robust in terms of axial drift because of its elongated PSF.These features are highly desirable in imaging studies that require long-term examination of dynamic features of biological specimens.Other 3D imaging modalities can also benefit from the E2E-BPF platform.For examples, optical coherence tomography 4,6,52,53 and photoacoustic 54 microscopy require high-resolution imaging over an extended DoF.Our BPF is expected to find its utility in enhancing the imaging performance and broadening its applications.

E2E-BPF design
The E2E-BPF is composed of K concentric rings, with their phase values parameterized by the vector, ϕ ¼ ðϕ 1 ; ϕ 2 ; ; ϕ K Þ.Each element of the vector ϕ can be initialized to an arbitrary value in the range of [−π, π], but is designed to converge to the binary states, i.e., 0 or π at the end of learning.To achieve this, a differentiable penalization function PðÁÞ was applied to ϕ.We conceived a penalization function given as: which exhibits the saddling points at (−π, 0, π).Note that this penalization function is the anti-derivative of the triple-well-potential function defined in 55 .With the penalized vector Φ, the E2E-BPF phase in the pupil plane can be expressed as: where ρ is the radial coordinate in the pupil plane that is normalized with NA=λ (0 ρ 1).

Imaging model
Consider a planar object I T , placed at a distance z from the focal plane of an imaging lens.The intermediate image Iðx 0 ; y 0 Þ obtained by the E2E-BPF microscope can be evaluated as the convolution of the object information with the depth-dependent PSF as: where denotes the convolution operation, and h x 0 ; y 0 ; Φ; ψ À Á is the PSF that results from BPF defined by Φ and defocus parameter ψ.The defocus parameter is related to the axial defocus distance z as 56 : where n medium denotes the refractive index of the medium.η is the noise, which is assumed to be additive Gaussian.
In our simulation, Gaussian noise with a standard deviation σ = 0.05 was applied to the normalized blurred image in the range of [0, 1].The depth-dependent PSF in an E2E-BPF microscope can be modeled as the squared magnitude of the Fourier transform of its pupil function: where F denotes the Fourier transform operator, exp Ài2πψρ 2 ð Þis the phase term from defocus, and the pupil function P ρ ð Þ is expressed as: Here, circðρÞ denotes a circular pupil with its radius normalized to NA=λ.

Loss function
The end-to-end loss (L E2E ) consists of the RMSE loss L RMSE and the BPF feature loss L BPF .First, the RMSE between two images is evaluated as: where NP is the number of pixels.
To enforce the phase values of BPF to the binary states during the learning stage, BPF feature loss L BPF and penalty factor α are introduced (see Eq. 2).The BPF feature loss function is given as: with where the multiplication operator in Eq. 11-b is an element-wise multiplication.Note that starting from a small positive value for the penalty parameter α, a gradient descent method was taken to minimize the loss function.Then, the penalty parameter was increased, and the process was repeated.Observe that, in the limit α ! 1, when the loss function is minimized, the penalty term converges to 0 for 10 epochs and the loss function is thereby minimized.Each epoch took ~1 h on a computer equipped with an Intel Xeon Gold 6226 R CPU and an NVIDIA RTX A6000 GPU.Over 10 epochs, the loss function progressively minimized.See Supplementary Section 4 for detailed information on the algorithm and the hyperparameters of the end-to-end network.

Image evaluation metric
The imaging performance of the E2E-BPF microscope was evaluated by computing its SSIM.SSIM is a wellknown quality metric used to measure the similarity between two images.The SSIM is defined as: where the μ and σ denote the mean intensity and standard deviation of an image, respectively.Note that σ I T Î is the covariance between I T and Î.The positive values of the SSIM index are in [0,1].A value of 0 indicates no correlation between the images, and 1 indicates that I T = Î.The regularization constants C 1 and C 2 are used to avoid a null denominator, and we set C 1 = 10 À4 and C 2 = 9 Á 10 À4 as used in [57].

Dataset
For the ground-truth datasets for training, histopathology images from the dataset 37 taken under a 60×/ 0.9NA microscope were used.The high-frequency features in the ground-truth image allowed physically accurate image degradation through a simulation of the E2E-BPF microscope (with or without BPF), primarily due to PSF convolution, defocus blur, and added noise.A total of 25,000 images were randomly assigned to the training, validation, and testing sets, which contained 22,000, 2200, and 820 images, respectively.During training, the images were scaled to fit the pixel size of the E2E-BPF microscope and augmented by rotation and flipping.

Experiment setup
The E2E-BPF microscope was built on an epi-fluorescence microscope composed of an objective lens (CFI Plan Apochromat Lambda 20×/0.75NA,Nikon, Japan) and a tube lens (TTL200, Thorlabs, USA).A 4-f optical setup (ACT508-180 & ACT508-300, Thorlabs, USA) relayed the image from the microscope onto the detector plane to achieve an effective magnification of 33.The E2E-BPF was placed in the conjugate plane of the back aperture of the objective lens.For excitation, light from a high-power broadband LED (SOLIC-3C, Thorlabs, USA) passed through an excitation filter (89013, Chroma, USA), and illuminated the specimen under the Köhler illumination condition.The fluorescence signal was collected by the objective lens, transmitted through a dichroic mirror, and imaged by a camera (Zyla 4.2, 4.2 MB format, 6.5 µm pixel size, Andor, U.K.) behind an emission filter.To enable imaging of a large specimen, lateral scanning was enabled by a pair of linear motorized stages (LNR502E/M, Thorlabs, USA), which featured a maximum travel range of 50 mm × 50 mm.Each image frame covered a FoV of 0.4 mm × 0.4 mm.

BPF fabrication
BPFs were fabricated on N-BK7 substrates using photolithography.This process enabled us to easily etch rings with lateral and depth uncertainties of a few micrometers and tens of nanometers, respectively.Under a monochromatic illumination at wavelength λ 0 , the desired etching depth was determined as: where n substrate is the refractive index of the material (in our case, SCHOTT N-BK7®) at λ 0 .For example, at λ 0 = 525 nm, d is obtained as 509 nm.In contrast, under multicolor illumination, the etching depth was determined at the center wavelength of emission.See Supplementary Section 11 for detailed information on the experimental set-up and fabricated E2E-BPF.

Sample preparation
Fluorescence microspheres (PS-Speck Microscope point source kit 7220, Molecular Probes, USA) with excitation/ emission wavelengths of 505/515 nm (green) were used to evaluate the imaging performance.The diameter of the microspheres was estimated as 0.175 ± 0.005 µm.A small drop of the microsphere solution was placed on a microscope slide and allowed to dry.After the sample was completely dried, a small drop of mounting medium was added, and a coverslip was placed on top of the medium.The edges of the coverslip were sealed.
We used a prepared slide of BPAE (FluoCells® prepared slide #1 (F36924) for single-color fluorescence imaging.The mitochondria of the cells were labeled with MitoTracker™ Red CMXRos.The stained cells were fixed and mounted on a glass slide using mounting medium.
A cryostat section of mouse kidney (FluoCells® prepared slide #3 (F24630), Molecular probes, USA) with a nominal thickness of 16 µm was used for large-scale tissue imaging.The tissue specimen was stained with a combination of fluorescent dyes.Alexa Fluor® 488 wheat germ agglutinin was used to label elements of the glomeruli and convoluted tubules.Filamentous actin prevalent in glomeruli and brush border was stained with red-fluorescent Alexa Fluor® 568 phalloidin.Nuclei were counterstained with the blue-fluorescent DNA stain DAPI.

Fig. 1
Fig.1Operating principle of E2E-BPF microscopy.An axi-symmetric BPF and image reconstruction network are jointly learned through the physics-informed neural network.The numerical phantom of an Arabidopsis thaliana in three-dimension space was considered.The learned BPF is fabricated and inserted in a pupil plane in an optical microscope, which produces projected volumetric image over the extended depth range.The acquired image is subsequently fed into the jointly learned reconstruction network to generate high-resolution, high-contrast image over the extended DoF.OBJ microscope objective; NA numerical aperture; TL tube lens; CAM camera

Fig. 2
Fig.2Learning pipelines for BPF and image reconstruction network.To begin, the phase filter is initialized with a continuous axi-symmetric function (e.g., axicon) and penalized by a nonlinear function that is designed to enforce the phase value in each ring to the binary states through the training process.The imaging model then predicts the image, which is then fed into the U-Net-based image reconstruction network to obtain the network output.This network output is compared against the ground-truth image, and optimization is performed to minimize the difference through a gradient-descent method

Fig. 3
Fig. 3 Numerical performance evaluation of E2E-BPF against clear and CPM pupil filters.The pictures of Lenna and mouse intestine tissue section were used as the reference, and numerically imaged by a microscope equipped with the filters.a-c Imaging results with clear, E2E-BPF, and CPM filters for the objects at various depth positions.Note that the results from E2E-BPF microscope and CPM were post-processed via the corresponding U-Nets optimized for each imaging condition.d RMSE and SSIM responses of each pupil filter as a function of defocus distance.The responses represent the mean RMSE and SSIM values evaluated over a randomly permuted test dataset.The solid lines represent the mean RMSE and SSIM values and the shaded areas represent standard error of the mean evaluated over randomly permuted test dataset (N = 820)

Fig. 4
Fig. 4 Imaging results of a fluorescent bead using standard and E2E-BPF microscopes.The top rows of a and b show the images of the fluorescent bead placed at various axial positions, and the bottom rows are the corresponding intensity profiles of the bead images, respectively.The red dots represent the raw data, and the solid curves are the results of Gaussian fitting.Scale bar in the images denotes 1 μm.c The graph shows the measured FWHMs of the PSFs for the standard and E2E-BPF microscopes.Each data point represents mean FWHM value, calculated from the measurements of 20 beads at 0.1 µm intervals along the depth axis.The error bars indicate standard deviation.For visual clarity, only every tenth data point is shown.It is evident that the E2E-BPF microscope provides an extended DoF while maintaining a PSF with the FWHM of 0.48 μm

z = - 9 Fig. 5
Fig. 5 Experimental imaging results of BPAE cells labeled with red fluorescent dyes at various depth locations using standard (clear aperture) and E2E-BPF microscopes.a E2E-BPF microscopy image of the BPAE cells at z= 0 μm.b, c Magnified images of the regions marked in (a) at various defocus distances, along with their SSIM values

Fig. 6
Fig. 6 E2E-BPF microscope for multicolor fluorescent imaging.a Whole slide image of the mouse kidney tissue section captured with the E2E-BPF microscope.Magnified images from the standard (b, d) and E2E-BPF (c, e) microscopes of the regions are marked with yellow dashed lines in (a).The E2E-BPF microscope could clearly visualize the structures across various depths, without serial refocusing